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BACKGROUND OF THE INVENTION 

The present invention relates generally to systems for allowing humans to interface with 
10 computer systems, and more particularly to methods for providing haptic feedback to the user 
interfacing with one or more computer applications involving sound output. 

up 

A user can interact with an environment displayed by a computer to perform functions 
!;f and tasks on the computer, such as playing a game, experiencing a simulation or virtual reality 
y environment, using a computer aided design system, operating a graphical user interface (GUI), 

navigate web pages, etc. Common human-computer interface devices used for such interaction 
''"^ include a mouse, joystick, trackball, gamepad, remote control, steering wheel, stylus, tablet, 
h== pressure-sensitive sphere, or the like, that is in communication with the computer system 

controlling the computer environment. The computer updates the environment in response to the 
1,0 user's manipulation of a physical manipulandum such as a joystick handle, button, or mouse, and 
2P provides visual and audio feedback to the user utilizing the display screen and audio speakers. 

The computer senses the user's manipulation of the user object through sensors provided on the 

interface device that send locative signals to the computer. 

In some interface devices, kinesthetic force feedback and/or tactile feedback is also 
provided to the user, more generally known collectively herein as "haptic feedback." These types 

25 of interface devices can provide physical sensations which are felt by the user manipulating a 
iiser manipulandum of the interface device. One or more motors or other actuators are coupled to 
the manipulandum and are connected to the controlling computer system. The computer system 
controls forces on the manipulandum in conjunction and coordinated with computer events and 
interactions by sending control signals or coinmands to the actuators. The computer system can 

30 thus convey physical force sensations to the user in conjunction with other supplied feedback as 
the user is grasping or contacting the interface device or manipulatable object of the interface 
device. 



The audio feedback provided to the user is an inherent part of many application programs, 
especially games. Some existing haptic feedback devices are designed to provide haptic 



sensations based directly on the sound output from the computer. The sound output waveform is 
directly routed to the interface device so that tactile sensations, such as vibrations, are based 
directly on the sound output waveform or a filtered portion thereof, much like the way a speaker 
operates. 

5 A disadvantage of existing haptic sensations based on direct sound waveforms is that the 

haptic sensations are simple effects based directly on the sound signals. No evaluation or 
processing of the soimd signals is accomplished before sending the signals to the haptic device. 
This can cause undesirable or disconcerting haptic sensations to be output to the user since not all 
of the sound output is appropriate for directly translation into haptic sensations. 

10 
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SUMMARY OF THE INVENTION 

The present invention is directed toward triggering the output of haptic sensations based 
on sound output from a computer device. The output of haptic sensations is inteUigently 
5 triggered by analyzing soxmd data for features, allowing an enhanced user experience. 

More particularly, an interface device of the present invention provides a method for 
triggering haptic sensations from sound features detected in sound output from a computer, 
where the haptic sensations are able to be output to a user of a haptic feedback device in 
communication with the computer. A portion of sound data is stored that is output to a user as 
10 audio from an application program running on the computer. The soimd data is stored in a 
memory buffer of the computer. The portion of sound data is analyzed using intelligent 

i.ri 

^^ heuristics to extract at least one sound feature from the portion of sound data. The execution of 
'■4 at least one haptic effect is triggered based on the sound feature(s), where the haptic effect is 
: commanded to the haptic feedback device approximately correlated to the output of the portion 
llSi of soimd to the user as audio. The haptic effect causes a haptic sensation to be output to the user. 

The triggered haptic sensation is preferably assigned to the sound features foimd in the 
sound data. Li some embodiments, analyzing the portion of sound data can includes processing 
the soimd data into multiple different frequency ranges and searching for sound features in each 
of the frequency ranges. A haptic effect can be triggered for each of the frequency ranges if the 

2Qi= sound feature is present in that frequency range. Filters can be applied to the sound data, or a fast 
Fourier transform can be used. Each frequency range can be associated with or mapped to a 
different haptic sensation. For example, each of the frequency ranges can be associated with a 
periodic haptic sensation having a different frequency. Other types of haptic sensations can also 
be mapped and assigned to sound features so that they will trigger upon output of those sound 

25 features. 

The present invention advantageously allows haptic feedback to be output by a computer 
system running an application program having sound output. The present invention intelligently 
assigns haptic sensations to features in the sound data to provide haptic feedback relevant to 
events in the application program which caused the sound output. This results in an overall 
30 improvement in user experience of the haptic feedback based on sound output. 

These and other advantages of the present invention will become apparent to those skilled 
in the art upon a reading of the following specification of the invention and a study of the several 
figures of the drawing. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



FIGURE 1 is a block diagram illustrating one embodiment of a haptic feedback system 
suitable for use with the present invention; 

FIGURE 2 is a side cross-sectional view of a mouse embodiment of the haptic feedback 
device suitable for use with the present invention; 

FIGURE 3 is a flow diagram illustrating a first embodiment of a method of the present 
invention for providing haptic effects to be output as haptic sensations based on provided sound 
data; 

FIGURE 4 is a flow diagram illustrating one embodiment for the step of Fig. 3 in which 
the sound data stored in the buffer is processed and analyzed; and 

FIGURE 5 is a flow diagram illustrating another embodiment for the step of Fig. 3 in 
which the sound data stored in the buffer is processed and analyzed. 



Docket No. IMM142 



4 



DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

FIGURE 1 is a block diagram illustrating a computer system 10 suitable for use with the 
present invention, including a haptic feedback interface device 12 in communication with a host 
5 computer 14. 

Host computer 14 preferably includes a host microprocessor 20, a clock 22, a display 
screen 26, and an audio output device 24. The host computer also includes other well known 
components, such as random access memory (RAM), read-only memory (ROM), and 
input/output (I/O) electronics (not shovm). The host computer 14 is a computing device that can 
10 take a wide variety of forms. For example, in the described embodiments computer 14 is a 
personal computer or workstation, such as a PC compatible computer or Macintosh personal 
computer, or a Sun or Silicon Graphics workstation. Such a computer 14 can operate under the 
Windows™ , MacOS™, Unix, MS-DOS, or other operating system. Altematively, host 
'"ll computer 14 can be one of a variety of home video game console systems commonly connected 
15-: to a television set or other display, such as systems available from Nintendo, Sega, Sony, or 
in Microsoft. In other embodiments, host computer 14 can be a "set top box", a "network-" or 
f " "internet-computer", a portable computer or game device, personal digital assistant (PDA), etc. 

is 

rj Host computer 14 preferably implements a host application program with which a user is 

■==f interacting via device 12 and other peripherals, if appropriate. In the context of the present 

20| invention, the host application program is a digital audio editing program, as described in greater 
detail below. Other application programs that utilize input of device 12 and output haptic 
feedback commands to the device 12 can also be used. The host application program preferably 
utilizes a graphical user interface (GUI) to present options to a user and receive input from the 
user. This application program may include the haptic feedback fimctionality described below; 

25 or, the haptic feedback control can be implemented in another program running on the host 
computer, such as a driver or other application program. Herein, computer 14 may be referred as 
providing a "graphical environment,", which can be a graphical user interface, game, simulation, 
or other visual environment. The computer displays "graphical objects" or "computer objects," 
which are not physical objects, but are logical software unit collections of data and/or procedures 

30 that may be displayed as images by computer 14 on display screen 26, as is well known to those 
skilled in the art. Suitable software drivers which interface software with haptic feedback 
devices are available from Immersion Corporation of San Jose, California. 

Display device 26 can be included in host computer system 14 and can be a standard 
display screen (LCD, CRT, flat panel, etc.), 3-D goggles, projection device, or any other visual 
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output device. Display device 26 displays images as commanded by an operating system 
application, simulation, game, etc. 

Audio output device 24, such as speakers, provides sound output to user. In the context 
of the present invention, other audio-related devices may also be coupled to the host computer, 
5 such as stereo receivers, amplifiers, etc. Other types of peripherals can also be coupled to host 
processor 20, such as storage devices (hard disk drive, CD ROM drive, floppy disk drive, etc.), 
printers, and other input and output devices. 

The haptic feedback interface device 12, such as a mouse, knob, gamepad, trackball, 
joystick, remote control unit, PDA screen, etc., is coupled to host computer 14 by a bi-directional 
10 bus 30. The bi-directional bus sends signals in either direction between host computer 14 and the 
interface device. Bus 30 can be a serial interface bus, such as an RS232 serial interface, RS-422, 
^ Universal Serial Bus (USB), MIDI, or other protocols well known to those skilled in the art; or a 
parallel bus or wireless link. Some interfaces can also provide power to the actuators of the 
device 12. 

1;|=„ Device 12 can include a local processor 40. Local processor 40 can optionally be 

,tl included within the housing of device 12 to allow efficient communication with other 
, components of the mouse. Processor 40 can be provided with software instructions to wait for 
f:i commands or requests from computer host 14, decode the command or request, and 
^ handle/control input and output signals according to the command or request. In addition, 
2&% processor 40 can operate independently of host computer 14 by reading sensor signals and 
calculating appropriate forces or commands fi-om those sensor signals, time signals, and stored or 
relayed instructions selected in accordance with a host command. Suitable microprocessors for 
use as local processor 40 include the MC68HC71 1E9 by Motorola, the PIC16C74 by Microchip, 
and the 82930AX by Intel Corp., for example, as well as more sophisticated force feedback 
25 processors such as the Immersion Touchsense Processor. Processor 40 can include one 
microprocessor chip, multiple processors and/or co-processor chips, and/or digital signal 
processor (DSP) capability. 

Microprocessor 40 can receive signals from sensor(s) 42 and provide signals to actuator 
44 in accordance with instructions provided by host computer 14 over bus 30. For example, in a 

30 local control embodiment, host computer 14 provides high level supervisory commands to 
processor 40 over bus 30, and processor 40 decodes the commands and manages low level force 
control loops to sensors and the actuator in accordance with the high level commands and 
independently of the host computer 14. This operation is described in greater detail in US 
Patents 5,739,81 1 and 5,734,373, both incorporated herein by reference. In the host control loop, 

35 force commands from the host computer instruct the processor to output a force or force 
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sensation having specified characteristics. The local processor 40 reports locative and other 
sensor data to the host computer which the host computer uses to update executed programs. In 
the local control loop, actuator signals are provided from the processor 40 to actuator 44 and 
sensor signals are provided from the sensor 42 and other input devices 48 to the processor 40. 
5 The processor 40 can process inputted sensor signals to determine appropriate output actuator 
signals by following stored instructions. Herein, the term "hap tic sensation" or "tactile 
sensation" refers to either a single force or a sequence of forces output by the actuator which 
provide a sensation to the user. The term "haptic effect" generally refers to the commands, 
parameters, and/or data sent to the device that define a haptic effect and which results in a haptic 
10 sensation when the effect is output as forces to the user by the device. 

Li yet other embodiments, other simpler hardware can be provided locally to device 12 to 
provide fimctionality as processor 40. For example, a hardware state machine or ASIC 
incorporating fixed logic can be used to provide signals to the actuator 44 and receive sensor 
'iCI signals from sensors 42, and to output tactile signals according to a predefined sequence, 
1 5= algorithm, or process. 
SI 

iJl In a different, host-controlled embodiment, host computer 14 can provide low-level force 

commands over bus 30, which are directly transmitted to the actuator 44 via processor 40. Host 
computer 14 thus directly controls and processes all signals to and from the device 12. In the 

^ simple host control embodiment, the signal from the host to the device can command the actuator 
201 to output a force at a predefined frequency and magnitude, or can include a magnitude and/or a 

Q direction, or be a simple command that indicates a desired force value to apply over time. 

Local memory 52, such as RAM and/or ROM, is preferably coupled to processor 40 in 
device 12 to store instructions for processor 40 and store temporary and other data. For example, 
force profiles can be stored in memory 52, such as a sequence of stored force values that can be 

25 output by the processor, or a look-up table of force values to be output based on the current 
position of the user object. In addition, a local clock 54 can be coupled to the processor 40 to 
provide timing data, similar to the system clock of host computer 14; the timing data might be 
required, for example, to compute forces output by actuator 44 (e.g., forces dependent on 
calculated velocities or other time dependent factors). In embodiments using the USB 

30 communication interface, timing data for processor 40 can be altematively retrieved from the 
USB signal. 

Sensors 42 sense the position or motion of the device and/or one or more manipulandums 
or controls and provides signals to processor 40 (or host 14) including information representative 
of the position or motion. Sensors suitable for detecting manipulation include digital optical 
35 encoders, optical sensor systems, linear optical encoders, potentiometers, optical sensors, 
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velocity sensors, acceleration sensors, strain gauge, or other types of sensors can also be used, 
and either relative or absolute sensors can be provided. Optional sensor interface 46 can be used 
to convert sensor signals to signals that can be interpreted by the processor 40 and/or host 14. 

Actuator(s) 44 outputs haptic sensations by transmitting forces to the housing or one or 
5 more manipulandums of the device 12 in response to signals received from processor 40 and/or 
host computer 14. Actuator 44 can be any of many types of actuators, including active actuators 
such as DC motors, voice coils, pneumatic or hydraulic actuators, torquers, piezoelectric 
actuators, moving magnet actuators, etc., or passive actuators such as brakes. 

Actuator interface 50 can be optionally connected between actuator 44 and processor 40 
10 to convert signals from processor 40 into signals appropriate to drive actuator 44. Interface 50 
can include power amplifiers, switches, digital to analog controllers (DACs), analog to digital 
Ql controllers (ADCs), and other components, as is well known to those skilled in the art. Other 
% input devices 48 are included in device 12 and send input signals to processor 40 or to host 14 
when manipulated by the user. Such input devices can include buttons, scroll wheels, d-pads, 
15'. dials, switches, or other controls or mechanisms. 

: * S 

;*[! Power supply 56 can optionally be included in device 12 coupled to actuator interface 50 

- and/or actuator 44 to provide electrical power to the actuator, or be provided as a separate 
component. Alternatively, power can be drawn from a power supply separate from device 12, or 
Q be received across the bus 30. Also, received power can be stored and regulated by device 12 

S3. 

1^1 and thus used when needed to drive actuator 44 or used in a supplementary fashion. Some 
\,:^ embodiments can use a power storage device in the device to ensure that peak forces can be 
applied (as described in U.S. Patent No. 5,929,607, incorporated herein by reference). 
Altematively, this technology can be employed in a wireless device, in which case battery power 
is used to drive the tactile actuators. A safety switch 58 can optionally be included to allow a 

25 user to deactivate actuator 44 for safety reasons. 

Many types of interface or control devices may be used with the present invention 
described herein. For example, such interface devices can include a haptic feedback trackball, 
joystick handle, steering wheel, knob, handheld remote control device, gamepad controller for 
video games or computer games, stylus, grip, wheel, button, cellular phone, PDA, touchpad, or 
30 other manipulatable object, surface, or housing. 

FIGURE 2 is a side cross-sectional view of a mouse embodiment 100 of device 12 for use 
with the present invention. 
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Mouse device 100 includes a housing 101, a sensing system 102, and an actuator 104. 
Housing 101 is shaped to fit the user's hand like a standard mouse while the user moves the 
mouse in the planar degrees of fi-eedom and manipulates the buttons 106. Other housing shapes 
can be provided in many different embodiments. 

5 Sensor 102 detects the position of the mouse in its planar degrees of fireedom, e.g. along 

the X and Y axes. In the described embodiment, sensor 102 includes a standard mouse ball 110 
for providing directional input to the computer system. Alternatively, an optical sensor or other 
type of sensor can be used. 

Mouse device 100 includes one or more actuators 104 for imparting haptic feedback such 
10 as tactile sensations to the user of the mouse. Actuator 104 is coupled to the housing 101 to 
provide haptic feedback to the user. In one embodiment, the actuator is coupled to an inertial 
i«| mass that is moved by the actuator. Inertial forces caused by the motion of the inertial mass are 
applied to the housing of the mouse w^ith respect to the inertial mass, thereby conveying haptic 
= -J feedback such as tactile sensations to the user who is contacting the housing. Some 
1;5; embodiments allow the actuator to move itself as the inertial mass. Such embodiments are 
iji described in greater detail in U.S. Patent No. 6,211,861 and U.S. Application No. 09/585,741, 
both incorporated herein by reference. Other types of interface devices, such as gamepads, 
handheld remote controls, cellular phones, PDA's, etc., can include such an actuator for inertial 

Ci tactile sensations. 

r\ 

2|E^f Other types of interface devices and actuators can also be used with the present invention. 

ilL For example, gamepads, mice, or other devices can include an eccentric rotating mass coupled to 
a rotating shaft of an actuator to provide inertial tactile sensations on the housing or 
manipulandum of the device. Other types of haptic devices can provide kinesthetic force 
feedback, such as joysticks, knobs, scroll wheels, gamepads, steering wheels, trackballs, mice, 

25 etc., in which forces are output in the sensed degree(s) of fireedom of a manipulandum. For 
example, embodiments of a kinesthetic mouse haptic device are disclosed in U.S. Patent Nos. 
6,100,874 and 6,166,723, both incorporated herein by reference in their entireties. 



Sound Output with Haptic Feedback 

30 The present invention improves the user experience in the output of haptic sensations 

coordinated with the sound output of one or more application programs (or other programs) running 
on the host computer. 
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The preferred embodiment of the present invention provides haptic effects for output to a 
haptic feedback device based on sound signals or data output by a host computer and which are 
output as haptic sensations to the user. In some alternate embodiments, the haptic effects can be 
similarly provided based on features in sound signals or data input to a computer system, e.g. from a 
5 microphone or audio device. Temporal features and characteristics of soxmd are identified and 
mapped to or associated with preprogrammed haptic effects. The haptic sensations based on these 
haptic effects can be rendered immediately on haptic feedback device 12, or can be stored in memory 
to form a "haptic profile" of a given sound. The method of the present invention is not to simply 
route sound signals directly to a haptic device for output as haptic sensations, as is done in the prior 
10 art, but to associate a variety of high-level sound features with a variety of high-level haptic 
sensations, using the soimd features to trigger the output of the appropriate haptic sensations. 

Q An important advantage of the present invention is that compelling haptic sensations are 

provided for any program that outputs sounds for a variety of events in that application program, e.g. 
\l games, without a software application developer having to expend effort in programming the haptic 
ISj effects in the program. Thus, any game program off the shelf can be easily made to work with haptic 
ijl feedback devices regardless of whether the game includes code to control haptic devices or whether 
'"^^ it was only developed for non-haptic-feedback devices, allowing the user a more compelling 
interactive and immersive experience with a wide variety of programs. Furthermore, the present 
^=1 invention can approximately distinguish between types of soimd effects and associate different haptic 
20-^ sensations with different events, allowing a more rich experience for the user. 

In one embodiment, a low-level driver program running on the host computer performs the 
methods of the present invention. In other embodiments, other levels of software running on the host 
(or on a microprocessor of the haptic feedback device) can perform some or all of the features or 
steps of the present invention (application program, API, etc.). Program instructions for 
25 implementing the present invention can be stored as hardware (e.g. logic components) or software 
stored on a computer readable medium, such as electrical memory, magnetic disk, optical disk, 
magnetic tape, etc. 

FIGURE 3 is a flow diagram illustrating a first embodiment 200 of a method of the present 
invention for providing haptic effects to be output as haptic sensations based on given sound data or 

30 sound signal. The method starts at 202, and in optional step 204, haptic effects are commanded to 
the device and not yet output to the user. For example, the haptic device 12 can include local 
memory that allows the device to store data for a number of different haptic effects. The haptic 
effects resident in the device memory can be played when a host command is received or based on 
other conditions (a predetermined time period has passed, etc.). The data sent to the device can 

35 include parameters for the haptic effects; e.g., for a periodic vibration effects, the parameters can 
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include a frequency, duration, and magnitude. The effects can be loaded to the device in step 204 
and in some embodiments can be commanded to play, e.g. at a magnitude of zero, in preparation to 
be commanded to higher magnitudes at a later step. Step 204 is most appropriate for those 
embodiments in which the same set of haptic sensations are always output to the user. In other 
5 embodiments, haptic effects can be commanded at the time they are to be output to the user. 

In next step 206, sound data is received from an application program or other software layer 
or program running on the host computer, or altemately, direction from an extemal audio device. 
The sound data is preferably digital data that can be in any standardized format, such as MIDI, digital 
sound samples in a wav file, an mp3 file, etc. The sound data is received in step 206 when it is also 
10 to be routed to an audio output device such as speakers and output to the user as sound. Typically, 
the sound data is generated by the application program from a sound event occurring in the 
application, such as a sound effect in a game resulting from game actions or events, music in a game 
"fl or other application program playing after a game starts or other event occurs, a beep in a graphical 
user interface interaction, the playing of music from a CD or DVD, or other sound event. In the 
lit described embodiment, the sound data received at this step to be analyzed is currently being output 
Si by speakers to the user. Thus, by the time the sound data is processed and haptic sensations output 
''b as described below, the sound data has aheady been output to the user. However, since a small 
j; amount of sound data is stored, only a delay of milliseconds occurs between sound and haptic output, 
^; which the user caimot discern. In other embodiments in which the sound data output is known in 
2^&l advance, haptic sensations can be output simultaneously, or close to simultaneously, with the sound 
data that triggered them. 

In step 208, a predetermined amount of the received soimd data is stored in a temporary 
buffer in the memory of the host computer 14, In one embodiment, this is an amount of data 
reasonably small enough to allow quick processing and associating with haptic sensations, while 
25 large enough to permit meaningfiil processing and analysis. In one embodiment, a portion of sound 
data equivalent to about 10 ms of play is stored, although different amoimts can be stored in other 
embodiments. For example, the sound can be digitally recorded in a 10 ms segment using 
DirectSound or other sound capture API on the host computer. 

In step 210, the stored sound data is processed and analyzed to identify particular sound 
30 characteristics or features which have been designated as relevant in the determination of which 
haptic sensations are to be output, i.e. these sound features are to act as cues to trigger the output of 
haptic sensations. In broad terms, the present invention uses intelligent heuristics to extract sound 
features that are likely to have meaning in an application program and to associate such sound 
features with predetermined, preprogrammed haptic sensations that correspond appropriately with 
35 the extracted sound features and the likely events these features represent. For example, likely 
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events in many current video game applications may be explosions from bombs, grenades, missiles, 
etc., the firing of a weapon such as a gun or rocket, the crash of a vehicle, the patter of footsteps, the 
sound of an object splashing into water, an alert sound for alerting the player to some game event, 
etc. These events may have particular sound features which can be recognized from the received 
5 sound data after appropriate analysis. 

A variety of different ways can be employed to process the data; some embodiments are 
described below with respect to Figs. 4 and 5.. These particular embodiments perform processing on 
the soimd data to divide the waveform represented by the soimd data into a number of different 
frequency ranges according to the present invention. Each frequency range of sound can then be 
analyzed separately for particular sound features. This frequency range division has the advantage 
that sounds often occur in different frequency ranges based on the event that triggered the sound; for 
example, an explosion in a game may have a low frequency sound while a gun shot may have a 
higher frequency sovmd. Types of events might therefore be approximately distinguished based on 
the frequency range in which the sound is located, allowing different haptic effects to be commanded 
and output as described below. 

Once the sound data has been processed and analyzed, in next step 212, haptic commands are 
output to the haptic device to cause haptic sensations to be output to the user manipulating the 
device. The haptic commands are triggered based on the sound features, if any, found in the sound 
data in step 210. Intelligent heuristics can be used to assign different haptic sensations to different 
types of soimds based, for example, on the frequencies of the sounds or other characteristics of the 
sounds. The haptic commands can cause device-resident haptic effects to start playing or play at a 
commanded magnitude. In one embodiment, the haptic effects that are already resident in memory 
on the device from step 204 are modified with new parameters or data as appropriate to implement 
the desired haptic sensations. Altematively, new haptic effects can be created and commanded to be 
output immediately by the device as haptic sensations. As explained above, the haptic sensations 
resulting from the commands may be output slightly after the associated sound featvu*es have been 
ou^ut as audio, but preferably soon enough so that the user cannot tell that there was a delay. 

In one embodiment, the commanded haptic sensations can be set to a magnitude proportional 
to the sound data magnitude in the associated frequency range. For example, embodiments discussed 
30 with reference to Figs. 4 and 5, the sound data is filtered or organized into 5 different frequency 
ranges. Each of these frequency ranges can have a different haptic sensation associated with it. For 
example, using an available inertial tactile feedback mouse, a periodic haptic effect of different 
frequency is associated with each of the frequency ranges. Thus, for example, a periodic haptic 
effect having a haptic frequency of 62 Hz (e.g., at or near a first resonance frequency of the haptic 
35 device to cause high magnitude sensations) is associated with the first sound frequency range (e.g., 0 
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Hz to 150 Hz). The other periodic haptic effects can have haptic frequencies of 75 Hz, 90 Hz, 115 
Hz, and 250 Hz, each corresponding to another sound frequency range in ascending order (other 
periodic frequencies can be used in other embodiments). The magnitudes of each of these periodic 
haptic effects can be set to a value proportional to the magnitude of the sound data in the associated 
5 frequency range. A running average of magnitude of ambient sound levels can be used as a 
minimum threshold for haptic effect generation, e.g. if the sovmd signal for a particular frequency 
range has a higher magnitude than the threshold, then a haptic effect associated with that frequency 
range can be commanded to be output or can be commanded to have a magnitude above a zero level. 

The method presented above thus varies the magnitudes of five different haptic effects based 
10 on the corresponding varying magnitudes in different frequency ranges of the soimd signal 
represented by the sound data. With some sounds that have components in all the frequency ranges, 
f-l this may cause five different haptic effects to be output at once. Hov^ever, since many sound features 
that are spread across several frequency ranges last for a short amount of time, the user perceives 
iJ^J only a short vibration or series of pulses rather than a mush of different frequencies. Other more 
l$l sustained sound effects may be concentrated in only one or two frequency ranges which will cause 
only one or two haptic effects to be output simultaneously (the haptic effects for the other frequency 
ill ranges will be set to zero magnitude). The user will thus feel a haptic sensations that roughly 
corresponds to the soimd effect's frequency. This is also discussed with respect to Fig. 4. 

i^n Thus, the amplitude of a haptic sensation that is associated with a particular frequency range 

2QD or band can correspond directly with the magnitude level of a sound signal in that frequency band, as 
i": explained above, la addition to this continuous haptic output, sudden spikes of amplitude in the 
sound signal can be mapped to additional jolts, pulses, or other short, strong haptic sensations which 
can be overlaid on the continuous vibration sensations. 

In other embodiments, more sophisticated haptic sensations can be output based on the soxmd 
25 data processing and analysis of step 210 and based on intelligent heuristics. In many applications 
such as games, the type of sound may be able to be determined based which frequency ranges it is 
primarily located. This allows a more sophisticated mapping scheme, where completely different 
haptic sensations can be mapped to different sound frequency ranges. This is also discussed with 
respect to Fig. 5. 

30 For example, it is often the case that weapon fire in a video game application is associated 

with a sudden burst of high frequency sound. Using this fact as an intelligent heuristic, in step 210 
the sound output data from the game can be analyzed and a feature that indicates that a high 
frequency sound has been output identified. For example, a particular frequency range can be 
checked for a sudden burst or spike in amplitude of sound. In step 212, a haptic sensation associated 

35 with weapon fire can be intelligently commanded to be output based on the sound burst found in the 
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sound. In some embodiments, the haptic sensation can be modified on the fly while the sound and 
the haptic sensations are playing, where the magnitude and duration of the haptic sensation is 
continually adjusted based on the magnitude of the sound burst above a predetermined threshold, the 
duration of the burst, and/or the specific frequency content of the burst. 

5 In some embodiments, a completely different haptic profile can be selected for a sound 

feature based on the specifics of the sound feature, where a pre-programmed mapping between sound 
characteristics and haptic effects is consulted (e.g., a look-up table or data table can be stored in 
memory). For example, a sound burst in a very different frequency range might trigger a completely 
different haptic sensation. A low frequency soimd burst in a video game might be, for example, the 
10 sound of a footstep of a character. An appropriate haptic sensation can be mapped to that soimd 
feature, such as a low- frequency vibration or imdulating force. 

Some games or other programs may output similar sound effects for similar events to allow 
approximate mapping as discussed above. Other games and programs, however, may have different 
SI events associated with similar-sounding soimds. Therefore, in one embodiment, a particular look-up 
ll: table or other mapping data can be associated with each individual game (or other application 
! f| program). Using the table, the method of the present invention can identify on the table which game 
is playing and then find the appropriate mapping between sound characteristics of that game and 
haptic sensations that have been assigned by the table's author. This allows for tuning of the 
Q heuristics tailored for specific games, enabling more accurate mappings. This can be advantageous 
2^^ to developers, who will not have to put haptic effects or mappings in their games themselves, but can 
iJ^I rely on the manufacturers of the haptic devices to put out driver updates including haptic mappings 
for games currently available based on the sound output. 

In alternate embodiments, the haptic effects that are associated with the found sound features 
are not immediately output as haptic sensations to the user, but are instead stored in memory to form 

25 a "haptic profile" of a given sound. This can be used to assist in the design of haptic effects. For 
example, a soimd file can be loaded and the user can select a control in a graphical user interface to 
cause the method of the present invention to generate a compound haptic effect that corresponds to 
the sound. The generated haptic effect (or haptic magnitudes, force values, etc.) can be then used in 
a program by commanding the generated compound haptic effect. Alternatively, the haptic profile 

30 (generated effect) can become a new haptic effect primitive to allow developers to create new effects 
based on this primitive, e.g. periodic effects, textures, temporal profiles, etc. can be created. 

For example, a game may have hundreds of sound effects. The method of the present 
invention can be instructed to load the game's sound files and create haptic effects from the sound 
data in a standardized format (such as .ifr files, used in protocols created by Immersion Corp. of San 
35 Jose, Califomia) in a few seconds. Thus, a developer need not create each haptic effect manually. 
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saving great amount of development time. A designer could also examine the generated haptic 
effects and edit them, if desired. This is advantageous to developers who wish to add code to their 
games to command haptic effects to haptic devices in the traditional way, but who do not wish to 
spend large amounts of time designing the appropriate haptic effects. 

5 After the output of the haptic sensation, the method retums to step 206 to check if further 

soimd data is received. If so, a next portion of sound data is processed as discussed above. If no 
fiirther sound data is received, the process is complete at 242 (to be started again the next time sound 
data is output). 

FIGURE 4 is a flow diagram illustrating one embodiment for step 210 of Fig. 3, in which the 
10 soimd data stored in the buffer is processed and analyzed. The method starts at 270, and in step 272, 
the stored sound data is set through multiple filters to isolate different frequency ranges. For 
Q example, to divide the sound data into five frequency ranges as described in the example above, a 
|^;f low-pass filter, a high-pass filter, and three bandpass filters can be employed. As one example, the 
\l low-pass filter can filter out all data in the fi-equencies outside 0-170 Hz, the first bandpass filter can 
ll; filter out all firequencies except 170-430 Hz, the second bandpass filter can filter out all fi-equencies 
1 fl except 430 Hz-2 kHz, the third bandpass filter can filter out all fi-equencies except 2-10 kHz, and 
the high-pass filter can filter out all fi*equencies lower than 10 kHz. 

In step 274, a running average of the sound magnitude firom each of the filter outputs is 
r:^! maintained. This allows ambient sounds output firom the computer to be averaged out so that only 
2(|J sounds louder than the ambient level will trigger haptic sensations. In next step 276, the magnitudes 
ul of the filter outputs are scaled to a range that is required for the haptic device 12 that the user is 
using. Thus, the actual filter outputs are not used to generate haptic sensations as in the prior art, but 
new haptic commands and effects are created based on the filter outputs. 

In some embodiments, the square of each filter's output magnitude can be scaled. This may 
25 be more effective in some embodiments to pick out only the larger peaks or spikes in each frequency 
range of sound data and reduce a sustained rumble/vibration of the device that may occiu: from too 
many peaks being detected. 

In next step 278, periodic haptic effects are assigned to each of the filter's outputs. The 
periodic effects can each be assigned a magnitude equal to the scaled magnitude found for each filter. 
30 If a filter has no output, i.e. no sound of that frequency range is present in the sound data, or if a 
filter's output is not above a predetermined threshold amplitude, then the haptic effect assigned to 
that frequency range would be assigned a magnitude of zero. The periodic sensations can each have 
one of five haptic frequencies found to be effective in conveying the corresponding sound frequency. 
Such periodic haptic sensations, such as vibrations, are quite appropriate for tactile devices such as 
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gamepads or mice providing inertial tactile sensations. The vibrations can also be output by 
kinesthetic feedback devices. Other types haptic sensations can also be output by kinesthetic device 
embodiments, having magnitudes based on the scaled magnitudes determined above. The type of 
haptic sensations assigned can be different in different embodiments, such as pops, jolts, springs, 
5 damping, etc., can also be output having magnitudes based on the scaled magnitudes; for example, a 
pre-programmed mapping can be consulted to assign a mapped haptic effect. 

The method is then complete at 280. The assigned periodic haptic sensations can be used in 
the next step 212 of Fig. 3, where haptic sensations are output top the user by commanding the haptic 
effects to the haptic feedback device. 

10 FIGURE 5 is a flow diagram illustrating a different embodiment 210' of step 210 of Fig. 3, in 

which the sound data stored in the buffer is processed and analyzed. The method starts at 290, and in 

Q step 292, a Fast Fourier Transform (FFT) is performed on the sound data to filter the data into 
different frequency components of the data's frequency spectrum. The frequency components each 

hi} 

\l span a frequency range based on the number of components output by the FFT and the entire 
l'l= frequency range covered. For example, if the FFT has 512 outputs, and the entire frequency range of 
I n the sound data is about 22 kHz, the each FFT output frequency component has a frequency range of 
iU 22 kHz / 512 = about 43 Hz. These frequency components can then be grouped to simi to desired 
f , frequency ranges. For example, to achieve the five frequency ranges similar to the embodiment of 
Q Fig. 4, four components can be combined to cover the first frequency range of 0 - 170 Hz, 6 
i& components cover the range 170-430 Hz, 37 components cover the range 430 Hz - 2 kHz, 185 
^ components cover the range 2 kHz - 10 kHz, and 256 components cover the range 10-22 kHz. 
Other frequency ranges and components can be used in other embodiments. 

In next step 294, the method checks whether any of the frequency ranges grouped in step 292 
have predetermined characteristics. These characteristics are features in the sound data that are 
25 significant enough to trigger the output of haptic sensations. In one embodiment, an average 
amplitude is maintained for each frequency range, and a spike in the amplitude is looked for. If a 
spike in amplitude of 3-4 times the average amplitude is found, then that is deemed significant and a 
haptic effect should be assigned to or triggered by that feature. 

If predetermined characteristics are found, then the process continues to step 296 to assign 
30 haptic effects to those frequency ranges having the predetermined characteristics, and the process is 
complete at 298 and the method retums to step 212 of Fig. 3. If in step 294 no frequency ranges are 
found to have the desired characteristics, then the process is complete at 298 and the method retums 
to step 212 of Fig. 3. 
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The haptic effects that are assigned to the frequency ranges having the desired characteristics 
can be any of several different types and/or have different parameters, and need not have a magnitude 
proportional to the amplitude of the corresponding sound data. For example, a jolt, pulse, or detent 
can be mapped to any extracted sound features (such as rise in amplitude). The magnitude of the jolt 
5 or detent can be based on the frequency range in which the amplitude spike was foimd, e.g. lower 
frequency ranges can be assigned higher magnitude haptic sensations (or vice-versa). Periodic 
sensations, directional sensations (for haptic devices able to output forces in particular directions in 
degrees of freedom of the device), spring sensations, damping sensations, etc. can be mapped to the 
different frequency ranges. In one embodiment, a haptic sensation can be increased in magnitude 
10 corresponding to the increased sound data amplitude, then the haptic amplitude can be ramped dovm 
automatically back to a zero level. Again, a pre-programmed mapping between sotmd features and 
haptic sensations can be consulted. 

The FFT outputs of the embodiment of Fig. 5 tend to be more efficient and provide more 
i,f| information than the filter outputs of the embodiment of Fig. 4. For example, FFT's can provide 
15f more detailed frequency information about where a spike in amplitude occurs, the phase offset of 
\i waves or features in the sound signal, etc. It should also be noted that the assigning or triggering of 

different types of haptic sensations described above for Fig. 5 can also be performed in the 
jl^" embodiment of Fig. 4, and the magnitudes of haptic sensations proportional to soimd amplitudes of 

Fig. 4 can be performed in the embodiment of Fig. 5. 

2Qf In some applications, the ambient background sounds, such as background music in game 

applications, can be ignored so that only sound effects in the game trigger haptic sensations to be 
output. Often, the sound effect magnitudes are much higher than the background music amplitudes, 
allowing the sound effects to be found as spikes above an average amplitude. In other 
circumstances, background music can ignored and sound effects processed by the type of audio data. 

25 For example, if the backgroimd music is in a MIDI format and the sound effects are .wav files, then 
MIDI data can be ignored and wav data processed by the present invention to provide haptic 
sensations. 

Some embodiments can also allow the user to enter preferences which can adjust how the 
sound data is analyzed, how the haptic effects are assigned, and how the haptic sensations are output. 
30 For example, haptic effect strength, the minimum threshold level of amplitude to recognize a sound 
feature (e.g., so that background music can be better ignored in a game), etc. 

The present invention can also be used in other types of application programs besides games. 
For example, a user's experience of streaming audio can be enhanced by adding haptic sensations to 
the audio output, so that if the user is contacting the haptic device while listening to the audio output, 
35 features in the audio such as music beats, sustained notes, features of speech etc. can be haptically 
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experienced by the user. Likewise, the present invention can be used with a sound track that is 
output with streaming video, e.g. over a network such as the Litemet. 

While this invention has been described in terms of several preferred embodiments, it is 
contemplated that alterations, permutations and equivalents thereof will become apparent to those 
skilled in the art upon a reading of the specification and study of the drawings. For example, many 
different embodiments of haptic feedback devices can be used to output the haptic sensations 
described herein, including joysticks, steering wheels, gamepads, and remote controls. Furthermore, 
certain terminology has been used for the purposes of descriptive clarity, and not to limit the present 
invention. 
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